Encapsulation and Its Discontents

Applying Normal Accident Theory to Software Design

July 23, 2008

CinematX Digital Inc.

Click left mouse button for next slide

photo © Iain Warde for openphoto.net CC:Attribution-ShareAlike

How to View the Slides

• Left mouse button advances to next slide, middle button goes back to the previous slide
• Move the mouse pointer to the lower right corner to display additional controls
• This slide show uses the S5 presentation system

The Tale of a Class

• Writing a management system for cloud computing
• The system operates as a Web service
• Main classes are fabrics (e.g. EC2) machines and commands
• Needed a "no-op" command, to verify that a machine is usable

It Starts Out Simple ...

... But Then

What If:

• I add logging?
• The behaviour is wrong in some cases (a bug)?
• etc., etc.
• This class is a PORV valve!

Normal Accident Theory

• Classic book: Normal Accidents by Charles Perrow (he developed the theory in 1979)
• Covers a wide range of systems
• Coupling and Interactive Complexity are the key measures
• Accidents in tightly coupled, complex systems are "normal"

Interaction and Coupling

From figure 3.1 of Perrow's Normal Accidents

The Parable of the PORV Valve

• Three Mile Island Nuclear accident
• The PORV valve caused unexpected interactions between two subsystems
• This confused the operators ...
• ... which ultimately caused loss of coolant in the reactor core

Famous Maxims

• "Flat is better than nested"

  Tim Peters

• "Strive for class interfaces that are complete and minimal"

  Scott Meyers

• "Favour object composition over class inheritance"

  Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides

Normal Accident Theory is an import Statement for Ideas

The PORV Valve as Anti-Pattern

• The component itself may be as good as it can be
• The problem is systemic
• Can try to resolve at design level ...
• ... but a refactoring may be in order

Reducing Interactive Complexity

Reducing Tight Coupling

Too Many Services Running?

• Long-lived processes have lots of opportunities for unintended behaviour
• Slow resource leaks, occasional corruption of internal data structures, etc. can lead to rare faults that are hard to understand
• Of course these are bugs, but Normal Accident Theory leads us to expect them!
• Making everything Web-based exacerbates the problem

Rediscovering the Command Line

• Great choice for an administrative facility -- reuses existing skills
• Applying new tools (e.g. Python) to an old idea results in a greatly simplified system
• Another promising direction is to use memcache as an I/O device
• Far less chance for unexpected interactions, because code runs for short bursts and then stops

IPython

• An interactive Python shell
• Also includes most traditional Unix shell features
• Provides lots of useful facilities, such as job control for tasks written in Python, and multiple hooks into its own command processing loop
• For example, in a few lines of code, you can automatically transform ordinary Python functions into command line utilities
• You have the option of later reusing these functions in a completely different environment, without IPython present

Conclusion

• Expect failures, and remember that systemic measures are likely needed to prevent accidents
• Use language features to reduce the interactive complexity of a system
• Techniques from earlier eras can help counteract the tendency towards tight coupling

Further Reading

• Normal Accidents by Charles Perrow (get the 1999 edition, it covers Y2K)
• The Next Catastrophe, Charles Perrow's recent book on this topic (it discusses the Internet specifically)
• Using Grok to Walk Like a Duck by Brandon C Rhodes (PyCon 2008 talk; a video is available)
• Using Optparse, Subprocess, and Doctest To Make Agile Unix Utilities by Noah Gift (PyCon 2008 talk)
• The IPython project and its completely redesigned successor IPython1 (the latter is still under development)

Thanks to Alex Martelli

For recommending Perrow's book, Normal Accidents, and pointing me in this fascinating direction.

Thanks to Our Brilliant Development Team

• Tom and Allison Taylor, our wonderful founders, who combine the deepest knowledge with absolute integrity
• Heath Johns, my co-author of formidable all-around intelligence, who unfortunately could not come
• Justin McMichael, who can write any program in any language (and is designing an electric car to outrace a Corvette)
• Aaron Mintz, our genius mathematician, who builds siege weaponry on the side
• Mike Edwards, who does remarkable work -- and even managed to get married
• Louis Blom, who keeps the entire company running smoothly. We must include him, since he can drink us under the table, and then beat all of us up